Determining fixation of a user&#39;s eyes from images of portions of the user&#39;s face enclosed by a head mounted display

ABSTRACT

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes multiple image capture devices positioned within the HMD to capture portions of a face of a user wearing the HMD. Images from an image capture device include a user&#39;s eye, while additional images from another image capture device include the user&#39;s other eye. The images and the additional images are provided to a controller, which applies a trained model to the images and the additional images to generate a vector identifying a position of the user&#39;s head and positions of the user&#39;s eye and fixation of each of the user&#39;s eyes. Additionally, illumination sources illuminating portions of the user&#39;s face include in the images and in the additional images are configured when the user wears the HMD to prevent over-saturation or under-saturation of the images and the additional images.

BACKGROUND

The present disclosure generally relates to head mounted displays, andmore specifically relates to determining a gaze of a user wearing a headmounted display.

Virtual reality systems typically include a display presenting contentto users. For example, many virtual reality, or augmented reality,systems include a head-mounted display including a display elementpresenting image or video data to a user. Content presented by thevirtual reality system depicts objects and users of the system.

Many virtual reality systems present graphical representations, oravatars, of users in a virtual environment to facilitate interactionsbetween users. However, conventional virtual reality systems providelimited graphical representations of a user. For example, avatarsrepresenting users in many conventional virtual reality systems have asingle facial expression, such as a default smiling or neutral facialexpression, or a limited set of facial expressions. These limited facialexpressions shown by avatars in virtual reality systems often presentusers from having a fully immersive experience in a virtual environment.

Tracking a user's face while the user interacts with a virtual realitysystem or an augmented reality system may provide a more immersiveinterface by allowing content presented by the virtual reality system oraugmented reality system to replicate movement of the user's face,providing a more immersive experience for the user. However,conventional facial tracking systems typically include a dedicatedperipheral, such as a camera, as well as markers positioned on the faceand body of a user being tracked. Using markers and the additionalperipheral may separate users from a provided virtual environment andare ill-suited for use in a portable, lightweight, and high-performancevirtual reality headset.

Additionally, including an eye tracking system in a head mounted displayused to present virtual reality or augmented reality content allowscontent presented by the head mounted display to provide more immersivecontent to a user wearing the head mounted display. For example, contentprovided to the user by the head mounted display is foveated, soportions of the content corresponding to a gaze direction of the user ispresented with a higher resolution than other portions of the presentedcontent. However, many conventional gaze tracking systems rely on highresolution images of a user's eyes, where a significant number of pixelsin captured images include the eyes of the user. Including image capturedevices dedicated to images of a user's eyes is often impractical forhead mounted displays that include other devices capturing informationabout a face of a user wearing the head mounted display.

SUMMARY

A virtual reality (VR) or augmented reality (AR) head mounted display(HMD) includes multiple image capture devices having complementaryfields of view and different depths. One or more of the image capturedevices are positioned to capture images of a portion of a user's faceexternal to a bottom side of the HMD. Additionally, one or moreadditional image capture devices are positioned to capture images ofother portions of the user's face within the HMD. In variousembodiments, a left image capture device is positioned within the HMDand proximate to a left side of the HMD and captures images of aportions of the user's face. A right image capture device is alsopositioned within the HMD and proximate to a right side of the HMD andcaptures portions of an additional portion of the user's face.Additionally, a central image capture device is positioned between exitpupils of the HMD that correspond to locations where the user's eyes arepositioned and captures images of a central portion of the user's face.Hence, the left image capture device, the right image capture device,and the central image capture device each capture images of portions ofthe user's face that are enclosed by the HMD.

In various embodiments, images captured by the left image capture deviceinclude the user's left eye, and additional images captured by the rightimage capture device include the user's right eye. The left imagecapture device and the right image capture device are coupled to acontroller that receives the images from the left image capture deviceand the additional images from the right image capture device. Thecontroller applies a trained model to an image and to an additionalimage that generates a vector describing a position of the head of theuser wearing the HMD. In various embodiments, the trained model is atrained convolutional neural network. Hence, the vector generated by thetrained model identifies fixation of the user's left eye and the user'sright eye relative to the position of the head of the user.

The trained model applied to the images and the additional images by thecontroller is trained based on data obtained from multiple users duringa calibration process and provided to the controller. During thecalibration process, the user wearing the HMD is presented with acalibration image via the HMD and instructed to direct the user's gazeto the calibration image. While continuing to direct the user's gaze tothe calibration image, the user repositions the user's head wheninstructed by the HMD. The left image capture device captures imagesincluding the user's left eye when the user's head has differentpositions. Similarly, the right image capture device captures additionalimages including the user's right eye when the user's head has differentpositions. Based on the images and additional images captured when theuser's head has different positions, gradient descent is applied to theimages and additional images captured when the user's head has differentpositions to generate a vector representing fixation of the user's gazerelative to the position of the user's head from one or more images andone or more additional images captured when the user's head has theposition. In various embodiments, the trained model is determined frommultiple users wearing different HMDs and refined for the user wearingthe HMD via the calibration process when the user wears the HMD. Thecontroller may modify content presented by the HMD based on the vectorgenerated by the trained model or may provide the vector to a console oranother device that generates content for presentation via a based onthe vector generated by the trained model.

Additionally, a left illumination source is positioned proximate to theleft image capture device, and a right illumination source is positionedproximate to the right image capture device. For example, the leftillumination source comprises one or more light emitting diodes (LEDs)positioned around a circumference of a lens of the left image capturedevice, while the right illumination source comprises one or more LEDspositioned around a circumference of a lends of the right image capturedevice. The left illumination source and the right illumination sourceemit light that illuminates the user's left eye and the user's righteye, respectively, and the left illumination source and the rightillumination source are coupled to the controller. For example, the leftillumination source and the right illumination source emit infraredlight, and the left image capture device and the right image capturedevice capture infrared light reflected by the user's left eye and bythe user's right eye, respectively.

To improve the images and the additional images captured by the leftimage capture device and by the right image capture device,respectively, the controller adjusts emission of light by the leftillumination source and by the right illumination source. In variousembodiments, the controller modifies light emission by the leftillumination source based on images received from the left image capturedevice and modifies light emission by the right illumination sourcebased on images received from the right image capture device. Forexample, the controller minimizes a function based on saturation orexposure by adjusting amounts of light emitted by different portions ofthe left illumination source or of the right illumination source. As anexample, the controller modifies an amount of light emitted by differentLEDs of the left illumination source (or of the right illuminationsource) based on minimization of the function. In some embodiments, thecontroller obtains information from a console or another sourcedescribing light emission by the left illumination source and the rightillumination source determined by other controllers and modifies theobtained information during a training process when the user is wearingthe HMD. This modification of the left illumination source and the rightillumination source based on images captured by the left image capturedevice and additional images captured by the right image capture device,respectively, allows the controller to prevent oversaturation orundersaturation of the images and the additional images by tailoringlight emission by the left illumination source or by the rightillumination source to the user wearing the HMD.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a virtual reality or an augmented realitysystem environment, in accordance with an embodiment.

FIG. 2 is a block diagram of a facial tracking system of the virtualreality or the augmented reality system, in accordance with anembodiment.

FIG. 3 is a wire diagram of a head mounted display, in accordance withan embodiment.

FIG. 4 is a rear view of the front rigid body of the HMD 300 shown inFIG. 3, in accordance with an embodiment.

FIG. 5 is a cross section of the front rigid body of the head mounteddisplay in FIG. 3, in accordance with an embodiment.

FIG. 6 is a flowchart of a method for determining fixation of a user'sleft eye and right eye from images of the user's face enclosed by a headmounted display (HMD) 105, in accordance with an embodiment.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

System Overview

FIG. 1 is a block diagram of a system environment 100 for providingvirtual reality (VR) content or augmented reality (AR) content inaccordance with an embodiment. The system environment 100 shown by FIG.1 comprises a head mounted display (HMD) 105, an imaging device 135, andan input/output (I/O) interface 140 that are each coupled to a console110. While FIG. 1 shows an example system environment 100 including oneHMD 105, one imaging device 135, and one I/O interface 140, in otherembodiments, any number of these components are included in the systemenvironment 100. For example, an embodiment includes multiple HMDs 105each having an associated I/O interface 140 and being monitored by oneor more imaging devices 135, with each HMD 105, I/O interface 140, andimaging device 135 communicating with the console 110. In alternativeconfigurations, different and/or additional components may be includedin the system environment 100.

The HMD 105 presents content to a user. Examples of content presented bythe HMD 105 include one or more images, video, audio, or somecombination thereof. In some embodiments, audio is presented via anexternal device (e.g., speakers and/or headphones) that receives audioinformation from the HMD 105, the console 110, or both, and presentsaudio data based on the audio information. An embodiment of the HMD 105is further described below in conjunction with FIGS. 3 and 4. In oneexample, the HMD 105 comprises one or more rigid bodies, which arerigidly or non-rigidly coupled to each other. A rigid coupling betweenrigid bodies causes the coupled rigid bodies to act as a single rigidentity. In contrast, a non-rigid coupling between rigid bodies allowsthe rigid bodies to move relative to each other.

The HMD 105 includes an electronic display 115, an optics block 118, oneor more locators 120, one or more position sensors 125, an inertialmeasurement unit (IMU) 130, and a facial tracking system 160. Theelectronic display 115 displays images to the user in accordance withdata received from the console 110. In various embodiments, theelectronic display 115 may comprise a single electronic display ormultiple electronic displays (e.g., a display for each eye of a user).Examples of the electronic display 115 include: a liquid crystal display(LCD), an organic light emitting diode (OLED) display, an active-matrixorganic light-emitting diode display (AMOLED), some other display, orsome combination thereof.

The optics block 118 magnifies received image light from the electronicdisplay 115, corrects optical errors associated with the image light,and presents the corrected image light to a user of the HMD 105. In anembodiment, the optics block 118 includes one or more optical elementsand/or combinations of different optical elements. For example, anoptical element is an aperture, a Fresnel lens, a convex lens, a concavelens, a filter, or any other suitable optical element that affects theimage light emitted from the electronic display 115. In someembodiments, one or more of the optical elements in the optics block 118may have one or more coatings, such as anti-reflective coatings.

Magnification and focusing of the image light by the optics block 118allows the electronic display 115 to be physically smaller, weigh less,and consume less power than larger displays. Additionally, magnificationmay increase a field of view of the displayed content. For example, thefield of view of the displayed content is such that the displayedcontent is presented using almost all (e.g., 110 degrees diagonal), andin some cases all, of the user's field of view. In some embodiments, theoptics block 118 is designed so its effective focal length is largerthan the spacing to the electronic display 115, which magnifies theimage light projected by the electronic display 115. Additionally, insome embodiments, the amount of magnification may be adjusted by addingor removing optical elements.

In an embodiment, the optics block 118 is designed to correct one ormore types of optical errors. Examples of optical errors include:two-dimensional optical errors, three-dimensional optical errors, orsome combination thereof. Two-dimensional errors are optical aberrationsthat occur in two dimensions. Example types of two-dimensional errorsinclude: barrel distortion, pincushion distortion, longitudinalchromatic aberration, transverse chromatic aberration, or any other typeof two-dimensional optical error. Three-dimensional errors are opticalerrors that occur in three dimensions. Example types ofthree-dimensional errors include spherical aberration, comaticaberration, field curvature, astigmatism, or any other type ofthree-dimensional optical error. In some embodiments, content providedto the electronic display 115 for display is pre-distorted, and theoptics block 118 corrects the distortion when it receives image lightfrom the electronic display 115 generated based on the content.

The HMD 105 may include various locators 120 in some embodiments. Thelocators 120 are objects located in specific positions on the HMD 105relative to one another and relative to a specific reference point onthe HMD 105. For example, a locator 120 is a light emitting diode (LED),a corner cube reflector, a reflective marker, a type of light sourcethat contrasts with an environment in which the HMD 105 operates, orsome combination thereof. In embodiments where the locators 120 areactive (i.e., an LED or other type of light emitting device), thelocators 120 may emit light in the visible band (i.e., ˜380 nm to 750nm), in the infrared (IR) band (i.e., ˜750 nm to 1 mm), in theultraviolet band (i.e., 10 nm to 380 nm), in some other portion of theelectromagnetic spectrum, or in some combination thereof.

In some embodiments, the locators 120 are located beneath an outersurface of the HMD 105, which is transparent to the wavelengths of lightemitted or reflected by the locators 120 or is thin enough not tosubstantially attenuate the wavelengths of light emitted or reflected bythe locators 120. Additionally, in some embodiments, the outer surfaceor other portions of the HMD 105 are opaque in the visible band ofwavelengths of light. Thus, the locators 120 may emit light in the IRband under an outer surface that is transparent in the IR band butopaque in the visible band.

The IMU 130 is an electronic device that generates fast calibration databased on measurement signals received from one or more of the positionsensors 125. A position sensor 125 generates one or more measurementsignals in response to motion of the HMD 105. Examples of positionsensors 125 include: one or more accelerometers, one or more gyroscopes,one or more magnetometers, another suitable type of sensor that detectsmotion, a type of sensor used for error correction of the IMU 130, orsome combination thereof. The position sensors 125 may be locatedexternal to the IMU 130, internal to the IMU 130, or some combinationthereof.

Based on the one or more measurement signals from one or more positionsensors 125, the IMU 130 generates fast calibration data indicating anestimated position of the HMD 105 relative to an initial position of theHMD 105. For example, the position sensors 125 include multipleaccelerometers to measure translational motion (forward/back, up/down,and left/right) and multiple gyroscopes to measure rotational motion(e.g., pitch, yaw, and roll). In some embodiments, the IMU 130 rapidlysamples the measurement signals and calculates the estimated position ofthe HMD 105 from the sampled data. For example, the IMU 130 integratesthe measurement signals received from the accelerometers over time toestimate a velocity vector and integrates the velocity vector over timeto determine an estimated position of a reference point on the HMD 105.Alternatively, the IMU 130 provides the sampled measurement signals tothe console 110, which determines the fast calibration data. Thereference point is a point describing the position of the HMD 105. Whilethe reference point may generally be defined as a point in space, inpractice, the reference point is defined as a point within the HMD 105(e.g., a center of the IMU 130).

The IMU 130 receives one or more calibration parameters from the console110. As further discussed below, the one or more calibration parametersare used to maintain tracking of the HMD 105. Based on a receivedcalibration parameter, the IMU 130 may adjust one or more IMU parameters(e.g., sample rate). In some embodiments, certain calibration parameterscause the IMU 130 to update an initial position of the reference pointso it corresponds to a next calibrated position of the reference point.Updating the initial position of the reference point as the nextcalibrated position of the reference point helps reduce accumulatederror associated with the determined estimated position. The accumulatederror, also referred to as drift error, causes the estimated position ofthe reference point to “drift” away from the actual position of thereference point over time.

The facial tracking system 160 generates reconstructions of portions ofa face of a user wearing the HMD 105, as further described below inconjunction with FIGS. 2-5. In an embodiment, the facial tracking system160 includes image capture devices, additional image capture devices,and a controller, as further described below in conjunction with FIG. 2.The facial tracking system 160 includes any suitable number of imagecapture devices or additional image capture devices in variousimplementations. In some embodiments, the facial tracking system 160also includes one or more illumination sources configured to illuminateportions of the user's face within fields of view of the one or more ofthe image capture devices or of the additional image capture devices.Based on images received from the image capture devices and from theadditional image capture devices, the controller generates a trainedmodel that maps positions of points identified within images captured bythe image capture devices and by the additional image capture devices toa set of animation parameters that map the positions of the identifiedpoints to a three dimensional model of a face presented via a virtualreality environment of the HMD 105. Additionally, based on images ofportions of the user's face enclosed by the HMD 105 that include theuser's left eye and the user's right eye, the facial tracking system 160determines a fixation of the user's left eye and a fixation of theuser's right eye relative to an orientation of the user's head.

The body tracking system 170 generates reconstructions of portions of abody of the user wearing the HMD 105. In an embodiment, the bodytracking system 170 includes imaging devices configured to captureimages of portions of the user's body outside of the HMD 105. Forexample, each imaging device is a camera having a field of viewsufficient to capture one or more portions of the user's body outside ofthe HMD 105. As an example, the body tracking system 170 comprisesmultiple video cameras positioned along a bottom surface of the HMD 105that are each configured to capture images including one or moreportions of the user's body (e.g., arms, legs, hands, etc.). In someembodiments, the body tracking system 170 also includes one or moreillumination sources configured to illuminate portions of the user'sbody within fields of view of the one or more of the imaging devices.The imaging devices are coupled to the controller of the facial trackingsystem, which generates a trained model that maps positions of pointsidentified within images captured by the imaging devices to a set ofbody animation parameters based on images received from the imagingdevices. The body animation parameters map positions of points of theuser's body identified from the images to a three dimensional model of abody presented via a virtual reality environment of the HMD 105.

The imaging device 135 generates slow calibration data in accordancewith calibration parameters received from the console 110. Slowcalibration data includes one or more images showing observed positionsof the locators 120 that are detectable by the imaging device 135. Insome embodiments, the imaging device 135 includes one or more cameras,one or more video cameras, any other device capable of capturing imagesincluding one or more of the locators 120, or some combination thereof.Additionally, the imaging device 135 may include one or more filters(e.g., used to increase signal to noise ratio). The imaging device 135is configured to detect light emitted or reflected from locators 120 ina field of view of the imaging device 135. In embodiments where thelocators 120 include passive elements (e.g., a retroreflector), theimaging device 135 may include a light source that illuminates some orall of the locators 120, which retro-reflect the light towards the lightsource in the imaging device 135. Slow calibration data is communicatedfrom the imaging device 135 to the console 110, and the imaging device135 receives one or more calibration parameters from the console 110 toadjust one or more imaging parameters (e.g., focal length, focus, framerate, ISO, sensor temperature, shutter speed, aperture, etc.).

The input/output (I/O) interface 140 is a device that allows a user tosend action requests to the console 110 and to receive responses fromthe console 110. An action request is a request to perform a particularaction. For example, an action request may be to start or end anapplication or to perform a particular action within the application.The I/O interface 140 may include one or more input devices. Exampleinput devices include: a keyboard, a mouse, a game controller, or anyother suitable device for receiving action requests and communicatingthe received action requests to the console 110. An action requestreceived by the I/O interface 140 is communicated to the console 110,which performs an action corresponding to the action request. In someembodiments, the I/O interface 140 may provide haptic feedback to theuser in accordance with instructions received from the console 110. Forexample, haptic feedback is provided when an action request is receivedor when the console 110 communicates instructions to the I/O interface140 causing the I/O interface 140 to generate haptic feedback when theconsole 110 performs an action.

The console 110 provides content to the HMD 105 for presentation to auser in accordance with information received from one or more of: theimaging device 135, the HMD 105, and the I/O interface 140. In theexample shown in FIG. 1, the console 110 includes an application store145, a tracking module 150, and a virtual reality (VR) engine 155. Someembodiments of the console 110 have different modules than thosedescribed in conjunction with FIG. 1. Similarly, the functions furtherdescribed below may be distributed among components of the console 110in a different manner than is described here.

The application store 145 stores one or more applications for executionby the console 110. An application is a group of instructions, that whenexecuted by a processor, generates content for presentation to the user.Content generated by an application may be in response to inputsreceived from the user via movement of the HMD 105 or the I/O interface140. Examples of applications include: gaming applications, conferencingapplications, video playback application, or other suitableapplications.

The tracking module 150 calibrates the system environment 100 using oneor more calibration parameters and may adjust one or more calibrationparameters to reduce error in determination of the position of the HMD105. For example, the tracking module 150 adjusts the focus of theimaging device 135 to obtain a more accurate position for observedlocators 120 on the HMD 105. Moreover, calibration performed by thetracking module 150 also accounts for information received from the IMU130. Additionally, if tracking of the HMD 105 is lost (e.g., the imagingdevice 135 loses line of sight of at least a threshold number of thelocators 120), the tracking module 140 re-calibrates some of or theentire system environment 100,

The tracking module 150 tracks movements of the HMD 105 using slowcalibration information from the imaging device 135. The tracking module150 determines positions of a reference point of the HMD 105 usingobserved locators 120 on the HMD 105 from the slow calibrationinformation and a model of the HMD 105. The tracking module 150 alsodetermines positions of a reference point of the HMD 105 using positioninformation from the fast calibration information. Additionally, in someembodiments, the tracking module 150 uses portions of the fastcalibration information, the slow calibration information, or somecombination thereof, to predict a future location of the HMD 105. Thetracking module 150 provides the estimated or predicted future positionof the HMD 105 to the engine 155.

The engine 155 executes applications within the system environment 100and receives position information, acceleration information, velocityinformation, predicted future positions, or some combination thereof ofthe HMD 105 from the tracking module 150. Based on the receivedinformation, the engine 155 determines content to provide to the HMD 105for presentation to a user. For example, if the received informationindicates that the user has looked to the left, the engine 155 generatescontent for the HMD 105 that mirrors the user's movement in a virtualenvironment. Additionally, the VR engine 155 performs an action withinan application executing on the console 110 in response to an actionrequest received from the I/O interface 140 and provides feedback to theuser that the action was performed. For example, the provided feedbackincludes visual or audible feedback via the HMD 105 or haptic feedbackvia the I/O interface 140.

Facial Tracking System

FIG. 2 is a block diagram of one embodiment of a facial tracking system160 of the system environment 100 for VR or AR. In the example shown inFIG. 2, the facial tracking system 160 includes one or more imagecapture devices 210, one or more additional image capture devices 215,and a controller 220. In other embodiments, different and/or additionalcomponents may be included in the facial tracking system 160.

The image capture devices 210 capture images of portions of a face of auser of the HMD 105, while the additional image capture devices 215capture additional images of other portions of the face of the user ofthe HMD 105. In various embodiments, the image capture devices 210 arepositioned so each image capture device 210 has a different field ofview and a different depth, so different image capture devices 210capture images of different portions of the user's face. Different imagecapture devices 210 have known positions relative to each other and arepositioned to have complementary fields of view including differentportions of the user's face. Similarly, the additional image capturedevices 215 are positioned so each additional image capture device 215has a different field of view and a different depth, so differentadditional image capture devices 215 capture different images ofdifferent portions of the user's face. Additionally, differentadditional image capture devices 215 have known positions relative toeach other and are positioned to have fields of view including differentportions of the user's face. The image capture devices 210 and theadditional image capture devices 215 are positioned relative to eachother to capture different portions of the user's face. For example, theimage capture devices 210 are positioned to capture portions of theuser's face that are outside of the HMD 105, such as lower portions ofthe user's face below a bottom surface of the HMD 105, while theadditional image capture devices 210 are positioned to captureadditional portions of the user's face that are enclosed by the HMD 105.FIG. 4 shows an example positioning of the image capture devices 210 andthe additional image capture devices 215.

Image capture devices 210 and additional image capture devices 215 maycapture images based on light having different wavelengths reflected bythe portions of the user's face. For example, image capture devices 210and additional image capture devices 215 capture infrared lightreflected by portions of the user's face. In another example imagecapture devices 210 and additional image capture devices 215 capturevisible light reflected by portions of the user's face. Image capturedevices 210 and additional image capture devices 215 have variousparameters such as focal length, focus, frame rate, ISO, sensortemperature, shutter speed, aperture, resolution, etc. In someembodiments, the image capture devices 210 and the additional imagecapture devices 215 have a high frame rate and high resolution. Theimage capture devices 210 and the additional image capture devices 215can capture two-dimensional images or three-dimensional images invarious embodiments.

In some embodiments, one or more illumination sources are coupled to oneor more surfaces of the HMD 105 and are positioned to illuminateportions of the user's face. Illumination sources may be positioned atdiscrete locations along the HMD 105. In some embodiments, the one ormore illumination sources are coupled to one or more exterior surfacesof the HMD 105. Additionally, one or more illumination sources may bepositioned within a rigid body of the HMD 105 to illuminate portions ofthe user's face enclosed by the rigid body of the HMD 105. Exampleillumination sources include be light-emitting diodes (LEDs) that emitlight in the visible band (i.e., ˜380 nm to 750 nm), in the infrared(IR) band (i.e., ˜750 nm to 1 mm), in the ultraviolet band (i.e., 10 nmto 380 nm), in some other portion of the electromagnetic spectrum, or insome combination thereof. In some embodiments, different illuminationsources have different characteristics. As an example, differentillumination sources emit light having different wavelengths ordifferent temporal coherences describing correlation between light wavesat different points in time. Further, light emitted by differentillumination sources may be modulated at different frequencies oramplitudes (i.e., varying intensity) or multiplexed in a time domain orin a frequency domain.

The controller 220 is coupled to the image capture devices 210 and tothe additional image capture devices 215 and communicates instructionsto the image capture devices 210 and to the additional image capturedevices 215. Instructions from the controller 220 to an image capturedevice 210 or to an additional image capture device 215 cause the imagecapture device 210 or the additional image capture device 215 to captureone or more images of portions of the user's face within the field ofview of the image capture device 210 or of the additional image capturedevice 215. In an embodiment, the controller 220 stores captured datadescribing characteristics of portions of the user's face (e.g., imagesof portions of the user's face) in a storage device accessible by thecontroller 220. The controller 220 includes a trained model that mapspositions of points identified within images captured by various imagecapture devices 210 or additional image capture devices 215 to a set ofanimation parameters that map points of the user's face included inimages captured by the image capture devices 210 or by the additionalimage capture devices 215 to a three dimensional (3D) model of a facethat is presented in a virtual reality environment or in an augmentedreality environment to present a graphical representation of the user'sface replicating the user's facial expression or facial movementcaptured by the image capture devices 210 or by the additional imagecapture devices 215. Additionally, the controller 220 includes anothertrained model that, when applied to images including portions of theuser's face including the user's left eye and other images includingportions of the user's face including the user's right eye, determinefixation of the user's left eye and of the user's right eye relative toa position of the user's head, as further described below in conjunctionwith FIG. 6.

In some embodiments, the controller 220 communicates the set ofanimation parameters to the console 110, which may store the facialanimation model in association with information identifying the user.The console 110 may communicate the set of animation parameters andinformation associated with the user to one or more other consoles 110,allowing HMDs 105 coupled to the other consoles 110 to present graphicalrepresentations of the user's face reflecting facial expressions orfacial movements of the user captured by the image capture devices 210and by the additional image capture devices 215. In some embodiments,the console 110 may communicate the set of animation parameters to aserver that stores animation parameters in association with informationidentifying different users. Additionally, the console 110 may modifycontent provided to the HMD 105 for presentation based on the set ofanimation parameters and other information received from the controller220, such as positions of points identified within images captured fromone or more image capture devices 210 or additional image capturedevices 215 and provided to the controller 220. For example, the console110 generates a graphical representation of the user's face that rendersmovement of the portions of the user's face on a three-dimensional modelbased on the set of animation parameters and positions of pointsidentified within captured images of portions of the user's face; thisallows the graphical representation of the user's face to replicateexpressions and movement of portions of the user's face captured by oneor more of the image capture devices 210 or by one or more of theadditional image capture devices 210.

Head Mounted Display

FIG. 3 is a wire diagram of one embodiment of a HMD 300. The HMD 300shown in FIG. 3 is an embodiment of the HMD 105 that includes a frontrigid body 305 and a band 310. The front rigid body 305 includes theelectronic display 115 (not shown in FIG. 3), the IMU 130, the one ormore position sensors 125, and the locators 120. In the embodiment shownby FIG. 3, the position sensors 125 are located within the IMU 130, andneither the IMU 130 nor the position sensors 125 are visible to theuser.

The locators 120 are located in fixed positions on the front rigid body305 relative to one another and relative to a reference point 315. Inthe example of FIG. 3, the reference point 315 is located at the centerof the IMU 130. Each of the locators 120 emit light that is detectableby the imaging device 135. Locators 120, or portions of locators 120,are located on a front side 320A, a top side 320B, a bottom side 320C, aright side 320D, and a left side 320E of the front rigid body 305 in theexample shown in FIG. 3.

In the example of FIG. 3, the HMD 300 includes image capture devices 210coupled to the bottom side 320A of the HMD 300. For example, an imagecapture device 210 is coupled to the bottom side 320C of the HMD 300proximate to the right side 320D of the HMD 300, and another imagecapture device 210 is coupled to the bottom side 320C of the HMD 300proximate to the left side 320E of the HMD 300. The image capturedevices 210 capture images of portions of the user's face below thebottom side 320C of the HMD 300. In the example of FIG. 3, the imagecapture device 210 captures images of portions of the user's faceproximate to the right side 320D of the HMD 300, while the other imagecapture device 210 captures images of portions of the user's faceproximate to the left side 320E of the HMD 300. While FIG. 3 shows anembodiment with two image capture devices 210, any number of imagecapture devices 210 may be included in various embodiments. The imagecapture devices 210 have specific positions relative to each other.Additionally, in various embodiments, different image capture devices210 have non-overlapping fields of view.

Similarly, a body tracking system 170 including multiple imaging devicesis coupled to the bottom side 320C of the HMD 300 in FIG. 3. Eachimaging device of the body tracking system 170 is configured to captureimages of portions of the user's body below the HMD 300 and external tothe HMD 300. In various embodiments, different imaging devices of thebody tracking system 170 have non-overlapping fields of view.

FIG. 4 is a rear view of the front rigid body 305 of the HMD 300 shownin FIG. 3. In the embodiment shown in FIG. 4, the front rigid body 305includes an eyecup assembly 435 including an exit pupil 420 and anadditional exit pupil 425. The exit pupil 420 is a position where an eyeof a user is positioned when the user is wearing the HMD 300, while theadditional exit pupil is a position where another eye of the user ispositioned when the user is wearing the HMD 300.

In the example of FIG. 4, a left image capture device 405 is coupled toan interior surface of the left side 320E of the front rigid body 305 ofthe HMD 300 and is proximate to a bottom side 320C of the front rigidbody 305 of the HMD 300. The left image capture device 405 capturesimages of a portion of the user's face. In the example of FIG. 4, theleft image capture device 405 captures images of a portion of the user'sface proximate to the to the left side 320E of the front rigid body 305of the HMD 300 and including an eye of the user positioned at the exitpupil 420 of the HMD 300. Additionally, a right image capture device 410is coupled to an interior surface of the right side 320D of the frontrigid body of the HMD 300 and is proximate to the bottom side 320C ofthe front rigid body 305 of the HMD 300. The right image capture device410 captures images of a portion of the user's face. In the example ofFIG. 4, the right image capture device 410 captures images of a portionof the user's face proximate to the to the right side 320D of the frontrigid body 305 of the HMD 300 and including an eye of the userpositioned at the additional exit pupil 425 of the HMD 300.

In various embodiments, a left illumination source is positionedproximate to the left image capture device 405 and a right illuminationsource is positioned proximate to the right image capture device 410.The left illumination source emits light illuminating the portion of theuser's face captured by the left image capture device 405, while theright illumination source emits light illuminating the additionalportion of the user's face captured by the right image capture device410. For example, the left illumination source and the rightillumination source each comprise one or more light emitting diodes(LEDs), although any suitable device emitting light may be used as theleft illumination source or the right illumination source. The leftillumination source may be a ring of one or more LEDs arranged around acircumference of a lens of the left image capture device 405, or theright illumination source may be a ring of one or more LEDs arrangedaround a circumference of a lens of the right illumination source 410 invarious embodiments. In various embodiments, the left illuminationsource and the right illumination source each emit infrared light toillumination the portion of the user's face and the additional portionof the user's face, respectively. However, in other embodiments, theleft illumination source and the right illumination source emit anysuitable wavelength or wavelengths of light to illuminate the portionand the additional portion of the user's face.

Additionally, the left illumination source is synchronized with the leftimage capture device 405, so the left illumination source illuminatesthe portion of the user's face when the left image capture device 405 iscapturing an image, but does not illuminate the portion of the user'sface when the left image capture device 405 is not capturing an image,in some embodiments. Similarly, the right illumination source issynchronized with the right image capture device 410, so the rightillumination source illuminates the additional portion of the user'sface when the right image capture device 410 is capturing an image, butdoes not illuminate the additional portion of the user's face when theright image capture device 410 is not capturing an image, in someembodiments. For example, the left image capture device 405 communicatesa control signal to the left illumination source when the left imagecapture device 405 captures an image, causing the left illuminationsource to emit light while the left image capture device 405 capturesthe image; similarly, the right image capture device 410 communicates acontrol signal to the right illumination source when the right imagecapture device 410 captures an image, causing the right illuminationsource to emit light while the right image capture device 410 capturesthe image. Alternatively, the left illumination source, the rightillumination source, or the left illumination source and the rightillumination source illuminate the portion of the user's face or theadditional portion of the user's face when the left image capture device405 or the right image capture device 410 capture images and when theleft image capture device 405 or the right image capture device 410 donot capture images.

The front rigid body 305 of the HMD 300 shown in FIG. 4 also includes acentral image capture device 415 that is positioned within the frontrigid body 305 between the exit pupil 420 and the additional exit pupil425. The central image capture device 415 is configured to captureimages of a central portion of the user's face that is enclosed by thefront rigid body 305. In various embodiments, the central portion of theuser's face includes a segment of the portion of the user's face as wellas a segment of the additional portion of the user's face. In someembodiments, the central image capture device 415 is coupled to theeyecup assembly 435 or is embedded in the eyecup assembly 435.Alternatively, the central image capture device 415 is coupled to aninterior surface of the front side 320A of the front rigid body 305. Theleft image capture device 405, the right image capture device 410, andthe central image capture device 415 are examples of the additionalimage capture devices 215 of the facial tracking system 160.

A central illumination source is positioned proximate to the centralimage capture device 415 in various embodiments. For example, thecentral illumination source comprises one or more light emitting diodespositioned around a circumference of a lens of the central image capturedevice 415, although the central illumination source may have anysuitable position in various embodiments. As described above, thecentral image capture device 415 provides a control signal to thecentral illumination source when the central image capture device 415 iscapturing an image, causing the central illumination source to emitlight while the central image capture device 415 is capturing an image,and provides an alternative control signal to the central illuminationsource when the central image capture device 415 stops capturing animage, causing the central illumination source to stop emitting lightwhen the central image capture device 415 is not capturing an image.Alternatively, the central illumination source is configured to emitlight both when the central image capture device 415 is capturing imagesand when the central image capture device 415 is not capturing images.

In the example of FIG. 4, an external image capture device 430 iscoupled to the bottom side 320C of the front rigid body 305 of the HMD300. The external image capture device 430 is configured to captureimages of a portion of the user's face external to the front rigid body305. For example, the external image capture device 430 is configured tocapture images of a portion of the user's face external to the bottomside 320C of the front rigid body 305 (e.g., a mouth of the user). Anexternal illumination source is positioned proximate to the externalimage capture device 430 in various embodiments. For example, theexternal illumination source comprises one or more light emitting diodespositioned around a circumference of a lens of the external imagecapture device 430, although the external illumination source may haveany suitable position in various embodiments. As described above, theexternal image capture device 430 provides a control signal to theexternal illumination source when the external image capture device 430is capturing an image, causing the external illumination source to emitlight while the external image capture device 430 is capturing an image,and provides an alternative control signal to the external illuminationsource when the external image capture device 430 stops capturing animage, causing the external illumination source to stop emitting lightwhen the external image capture device 430 is not capturing an image.Alternatively, the external illumination source is configured to emitlight both when the external image capture device 430 is capturingimages and when the external image capture device 430 is not capturingimages. The external image capture device 430 is an example of an imagecapture device 210 of the facial tracking system 160.

Additionally, the controller 220 or the console 110 providesinstructions to the left illumination source, the right illuminationsource, the central illumination source, and the external illuminationsource. Based on the instructions, the left illumination source, theright illumination source, the central illumination source, and theexternal illumination source modify emitted light.

In various embodiments, the left imaging device 405, the right imagingdevice 410, the central imaging device 415, and the external imagingdevice 430, each have a common field of view. For example, the leftimaging device 405, the right imaging device 410, the central imagingdevice 415, and the external imaging device 430 each have a field ofview of at least 105 degrees. In other embodiments, the left imagingdevice 405, the right imaging device 410, the central imaging device415, and the external imaging device 430 have one or more differentfields of view. For example, the left imaging device 405 and the rightimaging device 410 have narrower fields of view than the central imagingdevice 415. As another example, the external imaging device 430 has afield of view that is wider than fields of view of the left imagecapture device 405, the right image capture device, and the centralimage capture device 415.

FIG. 5 is a cross-sectional diagram of an embodiment of the front rigidbody 305 of the HMD 300 shown in FIG. 3. In the embodiment shown in FIG.5, the front rigid body 305 includes an eyecup assembly 500, an imagecapture device 210, an additional image capture device 215, a controller220, the body tracking system 170 an optics block 118, and an electronicdisplay 115. The image capture device 210 is coupled to a bottom side ofthe front rigid body 305 in the example shown by FIG. 5 and positionedto capture images of a portion 415 of the user's face. For purposes ofillustration, FIG. 5 shows a single image capture device 210; however,in various embodiments, any suitable number of image capture devices 210may be coupled to the front rigid body 305 and positioned to captureimages of the portion 515 of the user's face, as shown in the example ofFIG. 4. For example, the image capture device 210 is proximate to aright side of the front rigid body 305, while another image capturedevice 210 is proximate to a left side of the front rigid body 305, asshown in the example of FIG. 4. While FIG. 5 shows the image capturedevice 210 coupled to an exterior surface of the front rigid body 305,in some embodiments the image capture device 210 is coupled to aninterior surface of the front rigid body 305, which is transparent to ordoes not substantially attenuate wavelengths of light captured by theimage capture device 210.

Additionally, in the example of FIG. 5, the HMD 300 includes anadditional image capture device 215 within the front rigid body 305 andpositioned to capture images of a portion of the user's face enclosed bythe front rigid body 305. For purposes of illustration, FIG. 5 shows asingle additional image capture device 215; however, in variousembodiments, any suitable number of additional image capture devices 215may be coupled to or included in an interior surface of the front rigidbody 305 and positioned to capture images of one or more portions of theuser's face enclosed by the front rigid body 305. For example, theadditional image capture device 215 is proximate to a right side of aninterior of the front rigid body 305, while another additional imagecapture device 215 is proximate to a left side of the interior of thefront rigid body 305. While FIG. 5 shows the additional image capturedevice 215 coupled to an interior surface of the front rigid body 305,in some embodiments the additional image capture device 215 is includedin the front rigid body 305, which is transparent to or does notsubstantially attenuate wavelengths of light captured by the additionalimage capture device 215. Example positioning of one or more additionalimage capture devices is further described above in conjunction withFIG. 4.

The body tracking system 170 includes multiple imaging devicesconfigured to capture images of portions of the user's body. In theexample shown by FIG. 5, the body tracking system 170 is positioned on abottom side of the HMD 300, and imaging devices comprising the bodytracking system 170 are positioned to capture images of portions of theuser's body below the HMD 300. While FIG. 5 shows the body trackingsystem 170 coupled to an exterior surface of the front rigid body 305 ofthe HMD 300, in some embodiments the body tracking system 170 isincluded in the front rigid body 305, which is transparent to or doesnot substantially attenuate wavelengths of light captured by the imagingdevices of the body tracking system 170. The body tracking system 170 iscoupled to the controller 220, which generates graphical representationsof portions of the user's body included in images captured by the bodytracking system 170.

The front rigid body 305 includes an optical block 118 that magnifiesimage light from the electronic display 115, and in some embodiments,also corrects for one or more additional optical errors (e.g.,distortion, astigmatism, etc.) in the image light from the electronicdisplay 115. The optics block 118 directs the image light from theelectronic display 115 to a pupil 505 of the user's eye 510 by directingthe altered image light to an exit pupil of the front rigid body 305that is a location where the user's eye 510 is positioned when the userwears the HMD 300. For purposes of illustration, FIG. 5 shows a crosssection of the right side of the front rigid body 305 (from theperspective of the user) associated with a single eye 510, but anotheroptical block, separate from the optical block 118, provides alteredimage light to another eye (i.e., a left eye) of the user.

The controller 220 is communicatively coupled to the electronic display115, allowing the controller 220 to provide content for to theelectronic display 115 for presentation to the user (e.g., a graphicalrepresentation of one or more portions 515 of the user's face based ondata captured by the image capture device 210 or by the additional imagecapture device 215, a graphical representation of one or more portionsof the user's body included in images captured by the body trackingsystem 170). Additionally or alternatively, the controller 220 iscommunicatively coupled to the console 110 and communicates a set ofanimation parameters for generating graphical representations of one ormore portions 515 of the user's face or body to the console 110, whichincludes one or more graphical representations of portions 415 of theuser's face or body in content provided to the electronic display 115,or generates content for presentation by the electronic display 115based on the set of animation parameters received from the controller220. Additionally, the controller 220 is communicatively coupled to theimage capture device 210 and to the additional image capture device 215,allowing the controller 220 to provide instructions to the image capturedevice 210 and to the additional image capture device 215 for capturingimages of the portion 415 of the user's face or for capturing images ofan additional portion of the user's face, respectively. Similarly, thecontroller 220 is communicatively coupled to the body tracking system170, allowing the controller 220 to provide instructions to the bodytracking system 170 for capturing images of portions of the user's body.

Determining Fixation of a User's Eyes from Images of the User's FaceEnclosed by a HMD

FIG. 6 is a flowchart of one embodiment of a method for determiningfixation of a user's left eye and right eye from images of the user'sface enclosed by a head mounted display (HMD) 105. The method describedin conjunction with FIG. 6 may be performed by the facial trackingsystem 160, the console 110, or another system in various embodiments.Other entities perform some or all of the steps of the method in otherembodiments. Embodiments of the process may include different oradditional steps than those described in conjunction with FIG. 6.Additionally, in some embodiments, steps of the method may be performedin different orders than the order described in conjunction with FIG. 6.

As described above in conjunction with FIG. 4, the HMD 105 includes aleft image capture device 405 positioned within the HMD 105 (e.g.,within a front rigid body of the HMD 105) and proximate to a left sideof the HMD 105. The left image capture device 405 captures 605 images ofa portion of the user's face enclosed by the HMD 105 that includes theuser's left eye, as well as other features of the portion of the user'sface. Similarly, the HMD 105 includes a right image capture device 410positioned within the HMD 105 (e.g., within the front rigid body of theHMD 105) and proximate to a right side of the HMD 105. The right imagecapture device 410 captures 610 additional images of an additionalportion of the user's face enclosed by the HMD 105 that includes theuser's right eye, as well as other features of the additional portion ofthe user's face.

The left image capture device 405 and the right image capture device 410are each coupled to a controller, further described above in conjunctionwith FIG. 2. The controller 220 receives the images including the user'sleft eye from the left image capture device 405 and receives theadditional images including the user's right eye from the right imagecapture device 410. In various embodiments, timestamps or otherinformation identifying times when images were captured 405 and whenadditional images were captured 410 are received by the controller 220in conjunction with the images and the additional images. From theimages including the user's left eye and the additional images includingthe user's right eye, the controller 220 determines a fixation of theuser's left eye relative to a position of the user's head and a fixationof the user's right eye relative to the position of the user's head.Hence, the controller 220 determines where a gaze of the user's left eyeand a gaze of the user's right eye are directed.

To determine the fixation of the user's left eye and the fixation of theuser's right eye, the controller 220 applies 615 a trained model to animage and an additional image. Application of the model to the imagesand to the additional images generates 620 a vector indicating thefixation of the user's left eye and the fixation of the user's right eyerelative to the position of the head of the user. In variousembodiments, the controller 220 applies 615 the model to an image and toan additional image that are both associated with a common timestamp.The model is trained based on previously captured images of portions ofone or more other users' faces including the other users' left eyes andpreviously captured images of additional portions of the one or moreother users' faces including the other users' right eyes. one In variousembodiments, the trained model is a trained convolutional neural networkthat generates 620 the vector identifying fixation of the user's lefteye and fixation of the user's right eye relative to the position of thehead of the user. Unlike conventional gaze tracking systems, the imagesinclude the user's left eye and other features of the portion of theuser's face and the additional images include the user's right eye andother features of the additional portion of the user's face. This allowsthe controller 220 to determine fixation of the user's right eye andfixation of the user's left eye from images in which fewer pixels areused to represent the user's left eye and the user's right eye thanconventional gaze tracking systems.

In various embodiments, information used to train the trained model isobtained by the controller 220 from a console 130 or another device thatobtained information from other HMDs 105 worn by the other users duringa calibration process. For example, a calibration process is performedwhen a HMD 105 is worn by different users. During the calibrationprocess, the HMD 105 presents a calibration image to the user wearingthe HMD 105. The calibration image is presented at a fixed point on anelectronic display 115 of the HMD 105. For example, the presentedcalibration image comprises illumination of a specific set of pixels ata specific location of the electronic display 115 of the HMD 105. Whilethe calibration image is presented by the electronic display 115 of theHMD 105, the HMD 105 presents instructions directing the user to fix agaze of the user's left eye and to fix a gaze of the user's right eye tothe calibration image. While the gaze of the user's left eye and thegaze of the user's right eye are directed to the calibration image, theelectronic display 115 of the HMD 105 prompts the user to reposition theuser's head to specific positions at specific times. During a timeinterval after presenting a prompt to the user to reposition the user'shead to a specific position while maintaining fixation of the gaze ofthe user's left eye and fixation of the user's right eye on thecalibration image, the left image capture device 405 and the right imagecapture device 410 of the HMD 105 capture images and additional imagesof the user's face. The controller 220 receives the images and theadditional images captured while the user's head has the specificposition and associates information identifying the specific position ofthe user's head with the images and the additional images captured whenthe user's head has the specific position. Based on the images andadditional images captured when the user's head has different specificpositions, the controller 220, a console 130, or another device appliesgradient descent applied to the images and additional images associatedwith different specific positions of the user's head to train thetrained model to generate a vector representing fixation of the user'sgaze relative to the position of the user's head. In variousembodiments, the trained model is determined from images and additionalimages of multiple users wearing different HMDs 105 and refined for theuser wearing the HMD 105 by performing the calibration process when theuser wears the HMD 105. The trained model may account for additionalinformation, such as an interpupilary distance between centers of pupilsof the user's left eye and the user's right eye determined by thecontroller 220 using any suitable method. As another example, thetrained model accounts for light emitted by a left illumination sourceand by a right illumination source, further described above inconjunction with FIG. 4, and reflected by the user's left eye and by theuser's right eye, respectively. Additionally, the controller 220modifies the trained model over time based on images and additionalimages captured the left image capture device 405 and by the right imagecapture device 410, respectively, in some embodiments.

In some embodiments, the controller 220 modifies content presented bythe electronic display 115 of the HMD 105 based on the generated vector.For example, the controller 220 increases a resolution of a segment ofcontent at a location of the electronic display 115 of the HMD 105corresponding to the fixation of the user's left eye relative to aresolution of content presented at other locations of the electronicdisplay 115, and increases a resolution of a segment of content at anadditional location of the electronic display 115 of the HMD 105corresponding to the fixation of the user's right eye relative to aresolution of content presented at other locations of the electronicdisplay 115. Alternatively, the controller 220 transmits the generatedvector to a console 130 or another device that generates content forpresentation to the user via the electronic display 115 of the HMD 105.Subsequently, the console 130 or other device generates content forpresentation that accounts for the fixation of the user's left eye andthe fixation of the user's right eye.

In various embodiments, a left illumination source is positionedproximate to the left image capture device, and a right illuminationsource is positioned proximate to the right image capture device. Forexample, the left illumination source comprises one or more lightemitting diodes (LEDs) positioned around a circumference of a lens ofthe left image capture device, while the right illumination sourcecomprises one or more LEDs positioned around a circumference of a lendsof the right image capture device. The left illumination source and theright illumination source emit light that illuminates the portion of theuser's face within a field of view of the left image capture device 405and that illuminates the additional portion of the user's face within afield of view of the right image capture device 410. In variousembodiments, the left illumination source and the right illuminationsource both emit light having infrared wavelengths; however, the leftillumination source and the right illumination source may emit lighthaving any suitable wavelength or wavelengths in various embodiments.Light emitted by the left illumination source and by the rightillumination source allows the left image capture device 405 and theright image capture device 410, respectively, to capture further detailsof the portion of the user's face and of the additional portion of theuser's face.

The left illumination source and the right illumination source arecoupled to the controller 220, which provides instructions to the leftillumination source or to the right illumination source that modifieslight emitted by one or more portions of the left illumination source oremitted by one or more portions of the right illumination source. Invarious embodiments, the controller 220 modifies light emission by theleft illumination source based on one or more images received from theleft image capture device 405 and modifies light emission by the rightillumination source based on one or more images received from the rightimage capture device 405. In various embodiments, the controller 220receives one or more calibration images from the left image capturedevice 405 and modifies light emitted by one or more portions of theleft illumination source to minimize a function based on saturation orexposure of the one or more calibration images. Similarly, thecontroller 220 receives one or more additional calibration images fromthe right image capture device 410 and modifies light emitted by one ormore portions of the right illumination source to minimize the functionbased on saturation or exposure of the one or more additionalcalibration images. The controller 220 modifies light emitted bydifferent portions of the left image capture device 405 and emitted bydifferent portions of the right image capture device 410 so lightincident on the portion of the user's face and on the additional portionof the user's face optimizes the images and the additional imagescaptured by the left image capture device 405 and captured by the rightimage capture device 410, respectively. For example, the controller 220differently modifies light emitted by different LEDs comprising the leftimage capture device 405 to minimize the function based on saturation orexposure of the one or more calibration images. Similarly, thecontroller 220 differently modifies light emitted by different LEDscomprising the right image capture device 410 to minimize the functionbased on saturation or exposure of the one or more additionalcalibration images. Hence, modification of light emitted by the leftimage capture device 405 and by the right image capture device 410allows the controller to modify light incident on different regions ofthe portion of the user's face and on different regions of theadditional portion of the user's face.

The controller 220 may receive the calibration images and the additionalcalibration images during particular time intervals, or may select anysuitable images received from the left image capture device 405 and fromthe right image capture device 410 as the calibration images or the asthe additional calibration images, respectively. As another example, thecontroller 220 receives an indication that the HMD 105 is initially wornby the user and modifies light emitted by the left illumination sourceor by the right illumination source based on images received from theleft image capture device 405 and from the right illumination source 410within a threshold time after receiving the indication. For example, theindication is provided to the controller 220 from a position sensor 125in the HMD 105 has a specific change in orientation relative to areference orientation. As another example, the indication is provided tothe controller 220 from the electronic display 115 in response to theelectronic display receiving power.

In some embodiments, the controller 220 obtains information from aconsole 130 or another source describing light emitted by leftillumination sources 405 and right illumination sources 410 included inother HMDs 105. For example, the controller 220 obtains information fromthe console 130 describing light emission by left illumination sources405 and right illumination sources 410 included in other HMDs 105determined from training processes performed by the other HMDs 105. Thecontroller 220 subsequently modifies the obtained information based oncalibration images and additional calibration images captured by theleft image capture device 405 and by the right image capture device 405,as further described above. This modification of the left illuminationsource and the right illumination source based on images captured by theleft image capture device 405 and additional images captured by theright image capture device 410, respectively, allows the controller 220to prevent oversaturation or undersaturation of the images and theadditional images by tailoring light emission by portions of the leftillumination source or by portions of the right illumination source tothe user wearing the HMD 105, which improves an accuracy with which thecontroller 220 determines fixation of the user's left eye and the user'sright eye.

CONCLUSION

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

Embodiments disclosed herein may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,and any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto the viewer). Additionally, in some embodiments, artificial realitymay also be associated with applications, products, accessories,services, or some combination thereof, that are used to, e.g., createcontent in an artificial reality and/or are otherwise used in (e.g.,perform activities in) an artificial reality. The artificial realitysystem that provides the artificial reality content may be implementedon various platforms, including a head-mounted display (HMD) connectedto a host computer system, a standalone HMD, a mobile device orcomputing system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

Some portions of this description describe the embodiments in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are commonly used bythose skilled in the data processing arts to convey the substance oftheir work effectively to others skilled in the art. These operations,while described functionally, computationally, or logically, areunderstood to be implemented by computer programs or equivalentelectrical circuits, microcode, or the like. Furthermore, it has alsoproven convenient at times, to refer to these arrangements of operationsas modules, without loss of generality. The described operations andtheir associated modules may be embodied in software, firmware,hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, and/or it may comprise a general-purpose computingdevice selectively activated or reconfigured by a computer programstored in the computer. Such a computer program may be stored in anon-transitory, tangible computer readable storage medium, or any typeof media suitable for storing electronic instructions, which may becoupled to a computer system bus. Furthermore, any computing systemsreferred to in the specification may include a single processor or maybe architectures employing multiple processor designs for increasedcomputing capability.

Embodiments may also relate to a product that is produced by a computingprocess described herein. Such a product may comprise informationresulting from a computing process, where the information is stored on anon-transitory, tangible computer readable storage medium and mayinclude any embodiment of a computer program product or other datacombination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the patent rights. It istherefore intended that the scope of the patent rights be limited not bythis detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights, which is set forth in the following claims.

What is claimed is:
 1. A head mounted display (HMD) comprising: a rigidbody having a front side, a left side, a right side, a top side, and abottom side and including a display element configured to displaycontent to a user wearing the HMD and an optics block configured todirect light from the display element to an exit pupil and an additionalexit pupil of the HMD; a left image capture device coupled to aninterior surface of the HMD proximate to the left side of the frontrigid body and configured to capture images of a portion of the user'sface enclosed by the rigid body, the portion of the user's faceincluding a left eye of the user; a left illumination source positionedproximate to the left image capture device and configured to emit lightilluminating the portion of the user's face; an right image capturedevice coupled to an interior surface of the HMD proximate to the rightside of the front rigid body and proximate to the bottom side of therigid body and configured to capture images of an additional portion ofthe user's face enclosed by the rigid body, the additional portion ofthe user's face including a right eye of the user; a right illuminationsource positioned proximate to the right image capture device andconfigured to emit light illuminating the additional portion of theuser's face; and a controller coupled to the left image capture deviceand to the right image capture device, the controller configured to:obtain the images of the portion of the user's face including the lefteye of the user from the left image capture device; obtain the images ofthe additional portion of the user's face including the right eye of theuser from the right image capture device; generate a vector indicating afixation of the user's left eye and a fixation of the user's right eyerelative to the position of the head of the user by applying a model tothe images and to the additional images; modify content presented by anelectronic display included in the HMD based on the vector by increasinga resolution of a segment of content at a location of the electronicdisplay corresponding to the fixation of the user's left eye relative toa resolution of content presented at other locations of the electronicdisplay and increasing a resolution of an additional segment of contentat a location of the electronic display corresponding to the fixation ofthe user's right eye relative to the resolution of content presented atother locations of the electronic display.
 2. The HMD of claim 1,wherein the controller is further configured to: transmit the vector toa console, the console configured to generate content for presentationby an electronic display included in the HMD based on the fixation ofthe user's left eye and the fixation of the user's right eye.
 3. The HMDof claim 1, wherein the model comprises a trained convolutional neuralnetwork.
 4. The HMD of claim 3, wherein the trained convolutional neuralnetwork is trained based on application of a gradient descent process toimages of portion of other users' faces including left eyes of the otherusers obtained during a calibration process identifying a position ofthe other users' heads when the images of portion of other users' facesincluding left eyes of the other users were obtained and images ofportions of the other users' faces including right eyes of the otherusers obtained during the calibration process identifying a position ofthe other users' heads when the images of portion of other users' facesincluding right eyes of the other users were obtained.
 5. The HMD ofclaim 4, wherein the trained convolutional neural network is furthertrained based on application of the gradient descent process tocalibration images of the portion of the user's face including the lefteye of the user, the calibration images identifying a position of theuser's head when the calibration images were captured, and to additionalcalibration images of the additional portion of the user's faceincluding the right eye of the user, the additional calibration imagesidentifying a position of the user's head when the additionalcalibration images were captured.
 6. The HMD of claim 1, wherein thecontroller is further configured to modify light emitted by the leftillumination source based on one or more images captured by the leftillumination source; and modify light emitted by the right illuminationsource based on one or more images captured by the right illuminationsource.
 7. The HMD of claim 6, wherein the left illumination sourcecomprises a plurality of light emitting diodes (LEDs), and modify lightemitted by the left illumination source based on one or more imagescaptured by the left illumination source comprises: modifying lightemitted by at least a set of LEDs comprising the left illuminationsource to minimize a function based on saturation of the one or moreimages captured by the left illumination source.
 8. The HMD of claim 7,wherein the right illumination source comprises a plurality of lightemitting diodes (LEDs), and modify light emitted by the rightillumination source based on one or more images captured by the rightillumination source comprises: modifying light emitted by at least a setof LEDs comprising the right illumination source to minimize a functionbased on saturation of the one or more images captured by the rightillumination source.
 9. The HMD of claim 6, wherein modify light emittedby the left illumination source based on one or more images captured bythe left illumination source comprises: modify light emitted by one ormore portions of the left illumination source based on one or moreimages captured by the left illumination source in response to receivingan indication the HMD is initially worn by the user.
 10. The HMD ofclaim 9, wherein modify light emitted by the right illumination sourcebased on one or more images captured by the right illumination sourcecomprises: modify light emitted by one or more portions of the rightillumination source based on one or more images captured by the rightillumination source in response to receiving the indication the HMD isinitially worn by the user.
 11. The HMD of claim 1, wherein the leftillumination source comprises a plurality of light emitting diodes(LEDs) positioned around a circumference of a lens of the left imagecapture device.
 12. The HMD of claim 11, wherein the right illuminationsource comprises an additional plurality of light emitting diodes (LEDs)positioned around a circumference of a lens of the right image capturedevice.
 13. A method comprising: capturing images of a portion of auser's face enclosed by a head mounted display (HMD) via a left imagecapture device included in the HMD, the portion of the user's faceincluding a left eye of the user; capturing images of an additionalportion of the user's face enclosed by the head mounted display via aright image capture device included in the HMD, the additional portionof the user's face including a right eye of the user; applying a modelto the images and to the additional images, the model trained based onpreviously captured images including portions of users' faces includingleft eyes and previously captured images including portions of users'faces including right eyes; generating a vector indicating a fixation ofthe user's left eye and a fixation of the user's right eye relative to aposition of the head of the user from application of the model to theimages and to the additional images; and modify content presented by anelectronic display included in the HMD based on the vector to visuallydistinguish content at a location of the electronic displaycorresponding to the fixation of the user's left eye and content at alocation of the electronic display corresponding to the fixation of theuser's right eye relative to a resolution of content presented at otherlocations of the electronic display.
 14. The method of claim 13, whereincapturing images of the portion of the user's face enclosed by the HMDvia the left image capture device included in the HMD comprises:modifying light emitted by one or more portions of a left illuminationsource onto the portion of the user's face enclosed by the HMD based onone or more calibration images captured by the left image capturedevice; and capturing the images of the portion of the user's faceenclosed by the HMD via the left image capture device after modifyingthe light emitted by the one or more portions of the left illuminationsource.
 15. The method of claim 14, wherein the left illumination sourcecomprises a plurality of light emitting diodes (LEDs) positioned arounda circumference of a lens of the left image capture device and modifyinglight emitted by one or more portions of the left illumination sourceonto the portion of the user's face enclosed by the HMD based on one ormore calibration images captured by the left image capture devicecomprises: modifying light emitted by a LED positioned around thecircumference of the lens of the left image capture device to minimize afunction based on saturation of the one or more calibration images. 16.The method of claim 14, wherein capturing additional images of theadditional portion of the user's face enclosed by the HMD via the rightimage capture device included in the HMD comprises: modifying lightemitted by one or more portions of a right illumination source onto theadditional portion of the user's face enclosed by the HMD based on oneor more additional calibration images captured by the right imagecapture device; and capturing the additional images of the additionalportion of the user's face enclosed by the HMD via the right imagecapture device after modifying the light emitted by the one or moreportions of the right illumination source.
 17. The method of claim 16,wherein the right illumination source comprises a plurality ofadditional light emitting diodes (LEDs) positioned around acircumference of a lens of the right image capture device and modifyinglight emitted by one or more portions of the right illumination sourceonto the additional portion of the user's face enclosed by the HMD basedon one or more additional calibration images captured by the right imagecapture device comprises: modifying light emitted by an additional LEDpositioned around the circumference of the lens of the right imagecapture device to minimize the function based on saturation of the oneor more additional calibration images.